Generating High-Dynamic Range Images Using Varying Exposures

ABSTRACT

Systems and methods are provided for generating high-dynamic range images. In particular, a signal-to-noise (SNR) ratio objective associated with an imaging platform configured to capture one or more images of a region of interest can be determined. The SNR objective can specify a desired signal-to-noise ratio behavior as a function of a brightness of the region of interest. An exposure profile associated with the imaging platform can then be determined based at least in part on the SNR objective. Data indicative of a plurality of image frames, each depicting at least a portion of the region of interest, can then be obtained. The plurality of image frames being captured by the imaging platform in accordance with the exposure profile.

FIELD

The present disclosure relates generally to generating images, and moreparticularly to systems and methods for generating high-dynamic rangeimages.

BACKGROUND

Scenes captured, for instance, from overhead imaging platforms can havea high-dynamic range. For instance, details in one region of the scenecan be obscured in a shadow, while details in another region of thescene may be associated with a highly reflective rooftop. It can bedifficult to capture the entire dynamic range of a scene in an imagedepicting the scene. For instance, high light areas of the scene may besaturated, or the low light areas may be too noisy, or both.Conventional techniques for capturing high-dynamic range images includecombining multiple pictures taken at different exposures. For instanceexposure bracketing techniques can be used to capture high-dynamic rangeimages of a scene.

SUMMARY

Aspects and advantages of embodiments of the present disclosure will beset forth in part in the following description, or may be learned fromthe description, or may be learned through practice of the embodiments.

One example aspect of the present disclosure is directed to acomputer-implemented method of creating high-dynamic range images. Themethod includes determining, by one or more computing devices, asignal-to-noise ratio objective associated with an imaging platformconfigured to capture one or more images of a region of interest. Thesignal-to-noise ratio objective specifies a desired signal-to-noiseratio behavior as a function of a brightness of the region of interest.The method further includes determining, by the one or more computingdevices, an exposure profile associated with the imaging platform basedat least in part on the signal-to-noise ratio objective. The methodfurther includes obtaining, by the one or more computing devices, dataindicative of a plurality of image frames, each depicting at least aportion of the region of interest, the plurality of image frames beingcaptured by the imaging platform in accordance with the exposureprofile. The method further includes generating, by the one or morecomputing devices, a high-dynamic range image of at least a portion ofthe region of interest based at least in part on the plurality of imageframes.

Other example aspects of the present disclosure are directed to systems,apparatus, tangible, non-transitory computer-readable media, userinterfaces, memory devices, and electronic devices for creatinghigh-dynamic range images.

These and other features, aspects and advantages of various embodimentswill become better understood with reference to the followingdescription and appended claims. The accompanying drawings, which areincorporated in and constitute a part of this specification, illustrateembodiments of the present disclosure and, together with thedescription, serve to explain the related principles.

BRIEF DESCRIPTION OF THE DRAWINGS

Detailed discussion of embodiments directed to one of ordinary skill inthe art are set forth in the specification, which makes reference to theappended figures, in which:

FIG. 1 depicts an example imaging platform according to exampleembodiments of the present disclosure;

FIGS. 2-3 depict example imaging sensor configurations according toexample embodiments of the present disclosure;

FIG. 4 depicts an example image acquisition sequence according toexample embodiments of the present disclosure;

FIGS. 5-6 depict plots of example SNR responses according to exampleembodiments of the present disclosure;

FIGS. 7-8 depict plots of example SNR responses determined based on SNRobjectives according to example embodiments of the present disclosure;

FIG. 9 depicts a flow diagram of an example method of creatinghigh-dynamic range images by shaping an SNR curve according to exampleembodiments of the present disclosure;

FIG. 10 depicts a flow diagram of an example method of creatinghigh-dynamic range images according to example embodiments of thepresent disclosure; and

FIG. 11 depicts an example system according to example embodiments ofthe present disclosure.

DETAILED DESCRIPTION

Reference now will be made in detail to embodiments, one or moreexamples of which are illustrated in the drawings. Each example isprovided by way of explanation of the embodiments, not limitation of thepresent disclosure. In fact, it will be apparent to those skilled in theart that various modifications and variations can be made to theembodiments without departing from the scope or spirit of the presentdisclosure. For instance, features illustrated or described as part ofone embodiment can be used with another embodiment to yield a stillfurther embodiment. Thus, it is intended that aspects of the presentdisclosure cover such modifications and variations.

Example aspects of the present disclosure are directed to creatinghigh-dynamic images. For instance, a signal-to-noise (SNR) objective canbe determined. The SNR objective can be associated with an imagingplatform configured to capture one or more images of a region ofinterest. The SNR objective can specify a desired SNR curve relative toa brightness intensity associated with the region of interest. Anexposure profile can then be determined based at least in part on theSNR objective. A plurality of image frames can then be captured, eachdepicting at least a portion of the region of interest. For instance,the plurality of image frames can be captured by exposing an imagecapture device associated with the imaging platform in accordance withthe exposure profile. One or more of the captured image frames can thenbe combined to generate a high-dynamic range image of at least a portionof the region of interest.

More particularly, the imaging platform may be an overhead imagingplatform, such as a satellite, an airplane, a helicopter, an unmannedaerial vehicle (UAV), a drone, a balloon, etc. The imaging platform maybe configured to travel in a path over the region of interest called atrack. The path may include one or more straight lines or segments ormay be a curved path. For instance, the overhead imaging platform can beflown at a height over the region of interest. The imaging platform canbe configured to obtain a plurality of image samples or frames duringthe travel of the platform along the path. In some implementations, theimage frames can be captured in a continuous, rapid succession. Theimage frames can then be assembled into an output image on the groundvia digital processing.

In some implementations, the SNR objective can correspond to one or moreapplications associated with image capture by the imaging platform.Various imaging applications may require various SNR responsesassociated with the captured image frames. For instance, the SNRresponse may specify SNR as a function of the brightness of the scenecaptured by the images. In this manner, the SNR may be plotted as an SNRcurve relative to scene brightness. In some implementations, the scenebrightness can be associated with a scene reflectivity. Example SNRobjectives can include maintaining a substantially uniform SNR relativeto scene brightness. For instance, such SNR objective can includemaintaining the SNR within a predetermined SNR range. As another examplethe SNR objective can specify a minimum SNR that the SNR cannot fallbelow. For instance, a minimum SNR can be set for one or more brightnessranges.

Other example SNR objectives can include shaping the SNR curve inaccordance with a brightness distribution. For instance, the brightnessdistribution can correspond to distribution of brightness of a region ofinterest. In some implementations, the brightness can be a predeterminedbrightness distribution. For instance, the brightness distribution canbe an expected brightness distribution. In this manner, the brightnessdistribution may be determined based at least in part on a location ofthe imaging platform relative to the earth, a location of the sunrelative to the earth, an albedo associated with the region of interest,atmospheric absorption, an expected reflectance of one or more areaswithin the region of interest, expected atmospheric conditions and/orother suitable factors. In some implementations, the brightnessdistribution may be determined at least in part from one or moreadditional images depicting the region of interest. As another example,the brightness distribution can be determined and/or modifieddynamically as part of a feedback loop. For instance, one or more imagescaptured by the imaging platform can be analyzed to determine anexpected brightness distribution of one or more subsequent images to becaptured as part of the same image capturing session.

Once the SNR objective is determined, an exposure profile can bedetermined based at least in part on the SNR objective. The exposureprofile can include a plurality of exposure or integration times to beused in capturing image frames of the region of interest by the imagingplatform. In some implementations, the exposure profile can specify anumber of image frames to be captured and an integration time for eachimage frame to be captured. For instance, the number of frames to becaptured can be modified by adjusting a frame capture rate and/oradjusting a scanning rate. The integration times can be determined suchthat when a plurality of image frames are captured using the integrationtimes of the exposure profile, approximately corresponds to the SNRobjective. In this manner, the integration times can be selected toprovide a desired SNR behavior in the plurality of image frames. In someimplementations, the exposure profile may be determined based at leastin part on one or more image capture device(s). For instance, theexposure profile may be determined based at least in part on noisecharacteristics of the image capture device(s), dynamic range of theimage capture device(s), response linearity of the image capturedevice(s) and/or various other suitable factors associated with theimage capture device(s).

As indicated, the imaging platform can be configured to travel along apredetermined path above the region of interest. As the imaging platformtravels over the region of interest, the imaging platform can beconfigured to capture a plurality of image frames each depicting atleast a portion of the region of interest. For instance, the pluralityof image frames can be captured by one or more image capture devicesassociated with the imaging platform. For instance, the image capturedevice(s) may include one or more imaging sensors mounted to the imagingplatform. Such sensors may include line scan sensors, time delayintegration (TDI) sensors, two-dimensional (2D) staring sensors, colorwheel type 2D staring sensors, color filter array (CFA) sensors, and/orvarious other suitable sensors.

For instance, in implementations wherein a 2D staring sensor is used,the sensor can be configured to scan the region of interest and toacquire entire 2D frames taken as snapshots while the platform travelsalong the track. For instance, staring sensor imaging systems can bedesigned such that neighboring images contain overlapping measurementsof the region of interest. The presence of overlapping regions in theoutput images allows for later image processing to register neighboringimage frames and combine the images together to reconstruct an image ofthe region of interest.

In particular, each image frame can be acquired in accordance with thedetermined exposure profile. For instance, an image frame can becaptured by exposing the image capture device for the designatedintegration time as specified by the exposure profile. In someimplementations, each image frame can be captured using a differentintegration time. In implementations wherein successive image framescontain overlapping portions, each point on the ground in the region ofinterest may be captured in multiple image frames having differentintegration times. In this manner, the point as observed by multipleimage frames can have multiple associated SNRs. The plurality of imageframes captured in accordance with the exposure profile can then becombined to yield a desired SNR response (e.g. SNR curve). Inparticular, in some implementations, combining the image frames may beused to generate a high-dynamic range image having the desired SNRresponse.

According to other example aspects of the present disclosure, one ormore integration times can be determined for multiple portions of theimaging sensor. For instance, an SNR objective may be determined, and afirst integration time associated with a first portion of the imagesensor and a second integration time associated with a second portion ofthe image sensor can be determined based at least in part on the SNRobjective. A plurality of image frames can then be captured and at leasta portion of the image frames can be combined in accordance with the SNRobjective.

More particularly, in some implementations, the image capture device(s)associated with the imaging platform can include one or morepanchromatic components and one or more multispectral components. Forinstance, in some implementations, a single monolithic (or unitary)imaging sensor may be used. The single imaging sensor may include one ormore panchromatic blocks and one or more multispectral blocks. In suchinstances, the panchromatic and multispectral blocks may be implementedas filters disposed between the imaging sensor and the region ofinterest. For instance, the filters may be formed, mounted or otherwiseattached on a single, monolithic substrate.

In some implementations, the multispectral block may include one or morespectral filters configured to transmit light within a range ofwavelengths. For instance, the multispectral block may include a redspectral filter, a blue spectral filter, a green spectral filter, aninfrared spectral filter, a near infrared spectral filter, etc. Thepanchromatic filter(s) may have a broader spectral bandwidth than thespectral filters. For instance, in some implementations, thepanchromatic filter can have a bandwidth that is as large as two, three,or more (e.g., all) of the spectral filters.

In other implementations, the imaging sensor may include multiplephotosensors. For instance, the panchromatic filter(s) can be disposedover a first sensor, and the spectral filter(s) can be disposed over asecond sensor. In other implementations, a different sensor can be usedfor each spectral or panchromatic filter. The different imaging sensorscan, in some cases, have different pixel arrangements.

In such implementations wherein the imaging sensor(s) includepanchromatic and multispectral components, the integration times used tocapture images can be optimized based at least in part on thepanchromatic and/or multispectral components. Because the spectralfilters pass only a fraction of the total energy incident upon them, anexposure time that will saturate the panchromatic filter may notsaturate the spectral filters. In this manner, one or more integrationtimes can be selected such that the panchromatic filter improves thesignal in the darker regions of the scene, while saturating the highlights of the scene. The high lights may then be recovered by thespectral filters.

In example implementations, a single integration time may be selectedfor the entire sensor (e.g. the panchromatic block and the multispectralblock), or an integration time may be selected for each block and/orfilter. In implementations wherein each block has a differentintegration time, the blocks may be configured to expose simultaneouslyor at different times when capturing image frames. For instance, in someimplementations, the individual blocks may be exposed in a serialmanner. The integration times may be determined based at least in parton a desired SNR objective. For instance, as indicated above, theintegration times may be determined to shape an SNR curve relative toscene brightness.

As described above, a plurality of image frames depicting at least aportion of the region of interest can be captured. For instance, theimaging sensors can capture successive, overlapping snapshots of theregion of interest, such that each point in the region of interest iscaptured multiple times. In particular, each point may be captured oneor more times by the panchromatic portion of the imaging sensor, and oneor more times by the multispectral portion. In some implementations,each point may be captured one or more times by each spectral filter inthe spectral portion of the imaging sensor. Each image frame may becaptured by exposing the sensor(s) in accordance with the determinedintegration times. For instance, in implementations wherein themultispectral portion and the panchromatic portion of the imaging sensorhave different associated integration times, each portion may beconfigured to expose in accordance with the respective integrationtimes, either simultaneously or in a serial manner.

At least a portion of the captured image frames can then be combined inaccordance with the desired SNR objective. For instance, the imageframes can be combined to achieve a desired SNR response relative toscene brightness. In some implementations, the image frames may becombined to generate a high-dynamic range image.

In alternative implementations, a neutral density filter can be used. Aneutral density filter can be configured to reduce or modify theintensity of all wavelengths or colors of light passing through thefilter in an equal manner, giving no changes in hue of color rendition.The neutral density filter may be disposed between at least a portion ofthe imaging sensor(s) and the region of interest. For instance, theneutral density filter can be formed, mounted or otherwise attached tothe sensor(s). In example implementations, the neutral density filtercan be a fixed region neutral density filter or a graduated neutraldensity filter (e.g. light intensity varies across the surface). In someimplementations, multiple neutral density filters may be used having oneor more optical densities.

In such implementations, one or more integration times can be determinedbased at least in part on the neutral density filter. For instance, theone or more integration times can be determined based at least in parton the size of the neutral density filter, the position of the neutraldensity filter relative to the imaging sensor, the optical density ofthe neutral density filter, etc. When capturing images, each point inthe region of interest will be seen with full brightness in some frames,and reduced brightness in other frames (e.g. frames captured when theneutral density filter is disposed between the point and the sensor). Inthis manner, the integration time(s) can be determined such that brightareas in the region of interest will saturate in images captured by theportion of the sensor not having the neutral density filter, but darkareas will yield an improved SNR. The bright areas will then becorrectly captured when observed in the neutral density filter portionof the sensor.

It will be appreciated that various example aspects of the presentdisclosure can be implemented as standalone features, or in combinationwith one or more other example aspects of the present disclosure. Forinstance, an exposure profile can be determined for an imaging sensorhaving only a panchromatic portion, an imaging sensor having apanchromatic and multispectral portion, and/or an imaging sensor havinga neutral density filter in accordance with example embodiments of thepresent disclosure. As another example, the integration times determinedfor a multispectral portion and/or neutral density portion of an imagingsensor may be determined based at least in part on a desired SNRobjective. For instance, such integration times may be dynamicallygenerated or modified based at least in part on one or more currentimage frames captured by the imaging platform.

With reference now to the figures, example aspects of the presentdisclosure will be discussed in greater detail. For instance, FIG. 1depicts an example imaging platform 202 according to example embodimentsof the present disclosure. Imaging platform 202 may be configured to useone or more 2D staring sensors to acquire entire 2D frames taken assnapshots while the platform travels along a track 101 over a region ofinterest. In some implementations, imaging platform 202 can beconfigured such that neighboring images contain overlapping measurementsof the region of interest. For instance, the presence of overlappingregions in the output images allows for later image processing toregister neighboring image frames and mosaic the images together toreconstruct an image of the region of interest.

In particular, imaging platform 202 can acquire an entiretwo-dimensional image frame 203 in a single snapshot. Staring sensorscan be configured to capture images in rapid succession. For instance,an image can be captured sequentially through the capture or acquisitionof many different image frames (e.g. image frames 203, 204), each ofwhich can have some amount of overlap 205 with the image frames beforeand/or after it. In some implementations, the imaging region of astaring sensor may be thought of as a two-dimensional surface area.Light can be collected and bundled into individual pixels, whereby thenumber of pixels relative to the surface area of the image regiondetermines the resolution of the staring sensor. In various embodiments,the staring sensor can comprise a complementarymetal-oxide-semiconductor (CMOS) sensor or a charge coupled device (CCD)sensor. The staring sensor can include an array of photodiodes. In someembodiments, the staring sensor includes an active-pixel sensor (APS)comprising an integrated circuit containing an array of pixel sensors.Each pixel sensor can include a photodiode and an active amplifier. Forsome overhead imaging implementations, the staring sensor (and/or othercomponents of an overhead imaging platform) may be radiation hardened tomake it more resistant to damage from ionizing radiation in space.

As indicated, imaging platform 202 can be configured such thatneighboring image frames 203, 204 contain overlapping measurements ofthe region of interest (e.g., the overlap 205). In addition, asindicated above, each image frame can be captured using a predeterminedintegration time. For instance, in some implementations, an exposureprofile can be determined specifying an integration time associated witheach image frame to be captured. In other implementations, one or moreintegration times can be determined for one or more portions of theimaging sensor associated with imaging platform 202. The integrationtimes can be determined based at least in part on an SNR objective. Thepresence of overlapping regions in the output images allows for laterimage processing to register neighboring image frames and to combine theimages together to reconstruct a more accurate image of the region ofinterest. In some implementations, the images can be combined to shapean SNR curve to correspond to the SNR objective. In addition, bycombining many separate similar image frames together, the finalreconstructed image captured by a staring sensor can correct fordeviations in the motion of the imaging platform from the expecteddirection of travel 101, including deviations in speed and/or direction.

In some implementations, imaging platform 202 may further include acolor wheel sensor, a color filter array (CFA), such as a Bayer filter,a panchromatic channel (e.g. panchromatic filter or panchromaticsensor), one or more spectral channels (e.g. spectral sensor or spectralfilter), etc. For instance, imaging platform 202 may include an imagingsensor having a panchromatic block adjacent to a multispectral block. Inalternative implementations, imaging platform 202 can include aone-dimensional line sensor, such as a TDI sensor. A line scan sensorcan be a sensor having a single row of pixels for each color to becollected. The sensor is positioned in the platform so as to beperpendicular to the track direction thus moving in a linear manneracross a scene. Each row of pixels in an image is exposed in sequence asthe sensor moves across the scene, thus creating a complete 2D image. Anoverhead imaging platform that captures images with multispectral (e.g.,multiple color) information may use an independent line scan sensor foreach spectrum (e.g., color band) to be captured, wherein each line scansensor is fitted with a different spectral filter (e.g., color filter).

FIG. 2 depicts an example filter configuration for a two-dimensionalmultispectral staring sensor 300 that includes spectral filter strips305 a, 305 b, 305 c, and 305 d, according to example embodiments of thepresent disclosure. In particular, the sensor 300 includes a block 308of a plurality of spectral filter strips 305 a-305 d. In this example,the spectral filter strips 305 a-305 d are shaped in a long, narrowmanner spanning the axis or surface area of the staring sensor 300. Thespectral filter strips 305 can be disposed relative to the surface ofthe staring sensor 300 such that the filter strips 305 a-305 d aredisposed between the surface of the sensor and the region of interest tobe captured in an image. The region of interest may include, forexample, a portion of the surface of the earth that is to be imaged froman imaging platform. Light from the region of interest can pass throughthe filter strips 305 a-305 d before being detected by photosensitiveelements of the staring sensor. The strips 305 a-305 d may be formedover or on the staring sensor or may be attached or bonded to thestaring sensor. For example, the strips 305 a-305 d may be bonded to aceramic carrier or substrate for the staring sensor.

As shown in FIG. 2, the structure of the staring sensor 300 can bedescribed with reference to two perpendicular axes 306, 307, with theaxis 307 in the expected direction of travel 101 of the overheadplatform. For instance, the filter strips 305 a-305 d are orientedperpendicular to the axis 307 in the direction of travel 101. Each strip305 a-305 d can have a longitudinal axis that is oriented perpendicularto the axis 307 in the direction of travel 101 of the overhead platform.As will be further described below, each filter strip can have a heightin the direction 307. In some embodiments, the width of the filterstrips along the direction 306 (perpendicular to the direction of motion101) can be substantially the same as the length of the staring sensorin that direction, such that the filter strips substantially cover thesurface of the staring sensor 300.

In one example embodiment, the sensor comprises at least four spectralfilter strips (e.g., red, green blue, infrared). Other embodiments mayhave various other suitable numbers of filter strips. The spectralfilter strips 305 can be shaped roughly as rectangles (e.g., as shown inFIG. 2) or as parallelograms, squares, polygons, or any other suitableshape. In various embodiments, the filter strips cover substantially theentire surface of the staring sensor.

Each spectral filter strip can be configured to transmit light within arange of wavelengths. For example, a blue spectral filter strip can beconfigured to transmit wavelengths of light centered around the colorblue (e.g., 450-475 nm). Wavelengths of light outside the rangetransmitted by a filter are blocked, so that light outside thetransmitted range is not collected by the pixels of the staring sensorthat are “below” the filter strip. The range of wavelengths transmittedby each filter may vary. The range of wavelengths transmitted by aparticular filter strip may or may not overlap, at least partially, withthe range of wavelengths transmitted by other filter strips, dependingupon the embodiment. In addition to red (R), green (G), blue (B), andinfrared (IR) filters as illustrated, there are many other possiblewavelength ranges that may be transmitted by a spectral filter, forexample cyan, yellow, magenta, or orange. Infrared filters can includenear, mid, or far infrared filters. In some implementations, ultravioletfilters can be used. In some implementations, the wavelength ranges (orbandwidths) for the filter strips are selected to cover at least aportion of a desired spectral range, e.g., a visible spectral range, aninfrared spectral range, an ultraviolet spectral range, or a combinationof such spectral ranges. In some implementations, the spectral range ofthe filter strips 305 a-305 d is between about 450 nm to about 900 nm.In some implementations, the filter bandwidths can be less than about 20nm, less than about 50 nm, less than about 100 nm, less than about 150nm, less than about 200 nm, or some other range. Additionally, theordering of spectral filters (as well as placement in relation to apanchromatic sensor, if used) along the direction of relative motion 101is arbitrary, and as a consequence any order of the filter strips may beused.

In some embodiments, the height 307 of the filter strips along theirshort edges hfilter is between one and four times a minimum filterheight. In one embodiment, the minimum filter height can correspond tothe velocity of a point on the ground as seen by the sensor as it movesin the direction of travel 101, divided by a frame rate at which thestaring sensor 300 (and/or the imaging electronics such as controller320) captures image frames. In some embodiments, the controller 320 maybe integrated with the sensor 300, which may simplify packaging and usewith an imaging system. The controller 320 can be used or integratedwith any of the embodiments of the sensor 300 described herein toelectronically control image or video capture by the sensor 300.

Although FIG. 2 illustrates the staring sensor 300 having filter strips305 a-305 d that each have the same height 307, this is for purposes ofillustration and is not intended to be limiting. In otherimplementations, the heights of some or all of the filter strips can bedifferent from each other.

In addition to a two dimensional staring sensor, the sensor optionallymay also include a panchromatic block for capturing panchromatic imagedata in addition to the multispectral image data captured via the filterstrips 305 a-305 d of the staring sensor. The panchromatic block can besensitive to a wide bandwidth of light as compared to the bandwidth oflight transmitted by one or more of the spectral filter strips. Forexample, the panchromatic block may have a bandwidth that substantiallycovers at least a substantial portion of the combined bandwidths of thespectral filter strips. In various embodiments, the bandwidth of thepanchromatic block may be greater than about 50 nm, greater than about100 nm, greater than about 250 nm, or greater than about 500 nm. In oneimplementation, the bandwidth of the panchromatic block is between about450 nm and about 900 nm. In various implementations, the bandwidth ofthe panchromatic block can be greater than about two, greater than aboutthree, greater than about four, or greater than about five times thebandwidth of a spectral filter strip.

FIG. 3 depicts an example sensor including a two dimensional staringsensor 400 with spectral filter strips 505 a-505 d and a panchromaticblock 505 efgh, according to example embodiments of the presentdisclosure. As shown, the panchromatic block (505 efgh) is the samewidth (perpendicular to direction of relative motion 508) as eachindividual spectral filter strip (e.g., infrared 505 a, blue 505 b,green 505 c, red 505 d), but is four times the height 507 (parallel tothe direction of motion 101) of any of the individual spectral filterstrips 505 a-505 d. The height of the panchromatic block relative to theheight of the spectral filter strips may vary in variousimplementations. For instance, the width and height of the panchromaticblock may be determined based on the direction of relative motion 101 ofthe imaging platform, where height is parallel to the direction ofrelative motion 101, and width is perpendicular to the direction ofrelative motion 101.

FIG. 3 depicts gaps (e.g. 510 a) between the spectral filter strips andpanchromatic strips. It will be appreciated that such gaps may be anysuitable size. It will further be appreciated that, in someimplementations, such gaps may not be included at all. The total height506 of this embodiment of the staring sensor 400 can correspond to thesum of the heights of the panchromatic block 505 efgh, the spectralstrips 505 a-505 d, and the gaps 510 (if included).

Although the panchromatic block 505 efgh includes a single panchromaticfilter, it will be appreciated that, in some implementations, thepanchromatic block 505 efgh may include a plurality of panchromaticstrips having various suitable proportions. In some implementations, allportions of the spectral sensor(s) and panchromatic sensor(s) imagingareas may have the same pixel size, pixel shape, pixel grid or arrayplacement, and/or pixel density. However, in some cases individualportions of the sensors may have differing pixel sizes, shapes, grid orarray placements, and/or densities.

As indicated above, in some embodiments, an imaging sensor can include asingle (e.g., monolithic) two-dimensional staring sensor, wheredifferent portions of the sensor capture different data based on thespectral filters and/or panchromatic filters. In other embodiments,multiple staring sensors can be used. For example, the panchromaticstrip(s) can be disposed over a first photosensor, and the spectralfilter strip(s) can be disposed over a second photosensor. In otherimplementations, a different photosensor is used for each spectral orpanchromatic strip. The different photosensors can, in some cases, havedifferent pixel arrangements. In other embodiments, the staring sensormay be replaced by other types of spectral sensors such as line scanners(including TDI), color wheels, and CFA sensors.

FIG. 4 depicts an example imaging sequence 500 according to exampleembodiments of the present disclosure. As described herein, the sensor202 moves along the track direction, and thus moves relative to a regionof interest to be observed. The sensor captures image framessuccessively at a specified frame rate (frames per second or fps) and/orintegration time. As the sensor moves and captures image frames, everypoint in a region of interest is captured at least once by each spectralfilter (and/or panchromatic filter). In example implementations, thesensor movement may or may not be aligned with the motion of a movingoverhead platform.

As depicted in FIG. 4, images 409 a-409 h represent a sequence of eightsuccessive image frames captured by a sensor scanning over a region ofinterest. For instance, image sequence 500 can be captured using asensor corresponding to staring sensor 300 shown in FIG. 2. In thisexample, a panchromatic channel is not used. As shown, the sensorcaptures eight image frames, corresponding to capture times (CT) 1-8.The individual sensor captures are denoted by a capital letter for thefilter strip (from A to D) followed by a number for the capture time.For example, sensor capture D3 in image CT3 represents the capture bythe spectral strip D, 305 d, in the third capture time.

Each image frame can be captured by the image capture device using adesignated integration time. As described above, the integration timescan be determined based at least in part on an SNR objective. Forinstance, in some implementations, the integration times for eachcaptured image frame can be specified in an exposure profile. Theexposure profile can include predetermined integration times for eachimage frame to be captured. In this manner, the integration times can bechanged or varied over time to correspond to each successive capturetime. In some implementations, one or more integration times cancorrespond to one or more portions of the imaging sensor. For instance,each spectral strip can have a designated integration time. In thismanner, capturing image frames can include exposing each portion of theimaging sensor in accordance with the respective integration times. Inimplementations wherein the imaging sensor includes a panchromaticportion adjacent to the multispectral portion, the panchromatic blockcan also have one or more designated integration times.

In some implementations, after collection, all of the images can beco-registered. Once co-registration is completed, a separatereconstructed spectral image can be created for each color (spectral)band. The reconstructed color band images can be combined to create amultispectral image. In cases where the staring sensor includes apanchromatic sensor in addition to the multispectral sensor, capturedpanchromatic image data may be used to enhance the quality of amultispectral image. In some implementations, the images can be combinedbased at least in part on the SNR objective. For instance, the imagescan be combined in a manner that yields a desired SNR curve.

FIG. 5 depicts example SNR curves 602, 604 according to exampleembodiments of the present disclosure. In particular, SNR curves 602,604 provide a representation of signal-to-noise ratio as a function ofscene brightness (measured in digital numbers). SNR curve 602corresponds to a single image. SNR curve 604 corresponds to an SNR curveyielded from a combination of 16 image frames captured using identicalintegration times. FIG. 5 further depicts SNR curves 602, 604 overlaidagainst a PAN histogram 606 associated with a region of interest. Asshown by the histogram 606, the majority of the data captured by theimage capture devices corresponds to a low SNR. In this manner,increasing SNR in darker regions (as in SNR curve 604) can improve imagequality in the darker regions. As shown, SNR curve 602 provides a lowSNR in darker regions of an image. SNR curve 604 provides an increasedSNR relative to SNR curve 602. In this manner, SNR curve 604 may providea better SNR in darker regions of the image, but may provide an SNR thatis unnecessarily high in brighter regions.

As described above, multiple integration times can be combined to shapea SNR curve as a function of scene brightness. For instance, FIG. 6depicts an example SNR curve 608 corresponding to an image created bycombining multiple image frames. In particular, SNR curve 608 cancorrespond to an image created by combining an image frame captured at abaseline integration time (1× exposure) having an SNR curve 607, animage frame captured at twice the baseline integration time (2×exposure) having an SNR curve 609, and an image frame captured at fourtimes the baseline integration time (4× exposure) having an SNR curve611. As shown, SNR in an image is proportional to the square root of theintegration time use to capture the image. As further shown, dynamicrange of the image is inversely proportional to the length of theintegration time. In this manner, SNR associated with the combined imagecan be shaped based at least in part on the integration times used tocapture the image frames.

As described above, in some implementations, an SNR curve of a combinedimage can be shaped in accordance with a predetermined SNR objective.The SNR curve can be shaped based at least in part on the integrationtimes used to capture the individual image frames, and/or based at leastin part on the combination of the image frames. For instance, FIG. 7depicts a plot of an example SNR curve 610 according to exampleembodiments of the present disclosure. In particular, the SNR curve 610can be shaped in accordance with an SNR objective 612 of maintaining asubstantially uniform SNR. For instance, the SNR objective 612 caninclude maintaining the SNR within a threshold range for some range ofbrightness. In this manner, a plurality of integration times can bedetermined based at least in part on the SNR objective 612. Forinstance, as shown in FIG. 7, seven integration times (1×, 1×, 2×, 3×,5×, 8×, and 13×) are used to capture a plurality of image frames.

As another example, FIG. 8 depicts an SNR curve 614 determined inaccordance with an SNR objective 616 of approximately tracking abrightness distribution associated with a region of interest. Asdescribed above, in some implementations, the brightness distributioncan be predetermined based at least in part on location of the imagingplatform, location of the sun, expected brightness of the region ofinterest, etc. In some implementations, the brightness distribution canbe determined concurrently with the image acquisition, for instance, byanalyzing image frames captured in the current image acquisition. FIG. 8further depicts an SNR objective of maintaining the SNR above a minimumSNR for some range of brightness. In this manner, integration times canbe determined in accordance with multiple SNR objectives.

FIG. 9 depicts a flow diagram of an example method (700) of generatingimages according to example embodiments of the present disclosure.Method (700) can be implemented by one or more computing devices, suchas one or more of the computing devices depicted in FIG. 11. Inaddition, FIG. 9 depicts steps performed in a particular order forpurposes of illustration and discussion. Those of ordinary skill in theart, using the disclosures provided herein, will understand that thesteps of any of the methods discussed herein can be adapted, rearranged,expanded, omitted, or modified in various ways without deviating fromthe scope of the present disclosure.

At (702), method (700) can include determining an SNR objectiveassociated with an imaging platform. As described above, the SNRobjective can be associated with one or more desired SNR properties. Forinstance, the SNR objective can be to shape an SNR curve in accordancewith a brightness distribution, a minimum SNR, an SNR range, etc.

At (704), method (700) can include determining an exposure profileassociated with the imaging platform based at least in part on the SNRobjective. For instance, the exposure profile can include one or moreintegration times to be used in capturing a plurality of image frames ofa region of interest. In some implementations, the exposure profile canspecify an integration time for each image frame to be captured. In thismanner, the exposure profile can provide for integration times thatchange over time as the imaging platform captures image frames. In someimplementations, the exposure profile can be determined based at leastin part on one or more filters (e.g. spectral filters, neutral densityfilters, and/or panchromatic filters) associated with the image capturedevice. In some implementations an exposure profile can be determinedfor one or more portions of an imaging sensor. For instance, a firstexposure profile can be determined for a panchromatic portion of theimaging sensor, and a second exposure profile can be determined for amultispectral portion of the imaging sensor. In this manner, eachportion of the imaging sensor can be configured to capture image framesusing different integration times.

At (706), method (700) can include obtaining data indicative of aplurality of image frames. Each image frame can depict at least aportion of the region of interest. Each image frame can be captured bythe imaging platform in accordance with the exposure profile(s). Forinstance, capturing an image frame can include exposing at least aportion of the imaging sensor for the designated integration timespecified by the exposure profile. As indicated above, the image framesmay be captured in a successive manner, such that neighboring imageframes contain overlapping portions.

At (708), method (700) can include combining at least a portion of theimage frames to generate a high-dynamic range image. In someimplementations, the image frames can be co-registered prior tocombining. The image frames can be combined based at least in part onthe SNR objective. For instance, the image frames can be combined toyield a desired SNR behavior.

FIG. 10 depicts a flow diagram of an example method (800) of generatingan image according to example embodiments of the present disclosure. At(802), method (800) can include determining an SNR objective associatedwith an imaging platform. At (804), method (800) can include determininga first integration time associated with a first portion of an imagingsensor and a second integration time associated with a second portion ofthe imaging sensor. In some implementations, the first and secondintegration times can be determined based at least in part on the SNRobjective. In some implementations, the first and second integrationtimes can be determined based at least in part on one or more filtersassociated with the first and second portions of the imaging sensor. Forinstance, as described above, the first and second integration times canbe determined such that highlight areas associated with the region ofinterest will saturate in image frames captured by a panchromaticportion of the imaging sensor, but will not saturate in the spectralportions of the imaging sensor. As another example, the first and secondintegration times can be determined such that highlight areas associatedwith the region of interest will saturate in image frames captured by apanchromatic portion of the imaging sensor, but will not saturate in theneutral density portions of the imaging sensor. It will be appreciatedthat the first and second integration times may be the same integrationtimes, or may be different.

In some implementations, the first and second integration times can bedetermined as part of one or more exposure profiles. For instance, anexposure profile can be determined for each portion of the imagingsensor. The exposure profile can specify one or more integration timesto be used in capturing image frames.

At (806), method (800) can include obtaining data indicative of aplurality of image frames in accordance with the first and secondintegration times. For instance, an image frame can be captured by thefirst portion of the imaging sensor by exposing the first portion of theimaging sensor for the first integration time. An image frame can becaptured by the second portion of the imaging sensor by exposing thesecond portion of the imaging sensor for the second integration time. Invarious implementations the first and second portions of the imagingsensor can be exposed simultaneously or in a serial manner.

At (808), method (800) can include combining at least a portion of theimage frames to generate a high-dynamic range image. For instance asindicated above, the images can be combined based at least in part onthe SNR objective.

FIG. 11 depicts an example computing system 900 that can be used toimplement the methods and systems according to example aspects of thepresent disclosure. The system 900 can be implemented using aclient-server architecture that includes a server 910 that communicateswith one or more satellite systems 930 using radio frequencytransmission signals 940. The system 900 can be implemented using othersuitable architectures, such as a single computing device.

The system 900 includes a server 910. The server 910 can be associatedwith a control system for providing control commands to one or moresatellite systems 930. The server 910 can be implemented using anysuitable computing device(s). The server 910 can have one or moreprocessors 912 and one or more memory devices 914. The server 910 canalso include a communication interface used to communicate with one ormore satellite systems 930 over the network 940. The communicationinterface can include any suitable components for communicating with oneor more satellite systems 930, including for example, transmitters,receivers, ports, controllers, antennas, or other suitable components.

The one or more processors 912 can include any suitable processingdevice, such as a microprocessor, microcontroller, integrated circuit,logic device, or other suitable processing device. The one or morememory devices 914 can include one or more computer-readable media,including, but not limited to, non-transitory computer-readable media,RAM, ROM, hard drives, flash drives, or other memory devices. The one ormore memory devices 914 can store information accessible by the one ormore processors 912, including computer-readable instructions 916 thatcan be executed by the one or more processors 912. The instructions 916can be any set of instructions that when executed by the one or moreprocessors 912, cause the one or more processors 912 to performoperations. For instance, the instructions 916 can be executed by theone or more processors 912 to implement an SNR optimizer 920. SNRoptimizer 920 can be configured to determine an SNR objective associatedwith capturing a plurality of image frames depicting a region ofinterest, and determining one or more integration times to be used incapturing the image frames based at least in part on the SNR objective.

As shown in FIG. 11, the one or more memory devices 914 can also storedata 918 that can be retrieved, manipulated, created, or stored by theone or more processors 912. The data 918 can include, for instance,satellite data, imagery data, environmental data and other data. Thedata 918 can be stored in one or more databases. The one or moredatabases can be connected to the server 910 by a high bandwidth LAN orWAN, or can also be connected to server 910 through various othersuitable networks. The one or more databases can be split up so thatthey are located in multiple locales.

The server 910 can exchange data with one or more satellite systems 930using radio frequency transmissions signals 940. Although two satellitesystems 930 are illustrated in FIG. 11, any number of satellite systems930 can be configured to communicate the server 910. Each of thesatellite systems 930 can be any suitable type of satellite system,including satellites, mini-satellites, micro-satellites,nano-satellites, etc. In some implementations, satellite system 930 canbe an aircraft or other imaging platform such as an airplane, ahelicopter, an unmanned aerial vehicle, a drone, a balloon, or othersuitable aircraft.

Similar to the server 910, a satellite system 930 can include one ormore processor(s) 932 and a memory 934. The one or more processor(s) 932can include one or more central processing units (CPUs). The memory 934can include one or more computer-readable media and can storeinformation accessible by the one or more processors 932, includinginstructions 936 that can be executed by the one or more processors 932and data 938. For instance, the memory 934 can store instructions 936for implementing an image collector and a data transmitter configured tocapture a plurality of image frames and to transmit the plurality ofimage frames to a remote computing device (e.g. server 910). In someimplementations, aspects of SNR optimizer 920 may be implemented withinsatellite system 930. For instance, various aspects of SNR optimizer 920may be implemented within satellite system 930 instead of or in additionto various aspects of SNR optimizer 920 being implemented within server910.

The satellite system 930 can also include a communication interface usedto communicate with one or more remote computing devices (e.g. server910) using radio frequency transmission signals 940. The communicationinterface can include any suitable components for interfacing with onemore remote computing devices, including for example, transmitters,receivers, ports, controllers, antennas, or other suitable components.

In some implementations, one or more aspects of communication amongsatellite system 930, server 910, and/or a ground communication stationmay involve communication through a network. In such implementations,the network can be any type of communications network, such as a localarea network (e.g. intranet), wide area network (e.g. Internet),cellular network, or some combination thereof. The network can alsoinclude a direct connection between a satellite system 930 and theserver 910. In general, communication through the network can be carriedvia a network interface using any type of wired and/or wirelessconnection, using a variety of communication protocols (e.g. TCP/IP,HTTP, SMTP, FTP), encodings or formats (e.g. HTML, XML), and/orprotection schemes (e.g. VPN, secure HTTP, SSL).

The technology discussed herein makes reference to servers, databases,software applications, and other computer-based systems, as well asactions taken and information sent to and from such systems. One ofordinary skill in the art will recognize that the inherent flexibilityof computer-based systems allows for a great variety of possibleconfigurations, combinations, and divisions of tasks and functionalitybetween and among components. For instance, server processes discussedherein may be implemented using a single server or multiple serversworking in combination. Databases and applications may be implemented ona single system or distributed across multiple systems. Distributedcomponents may operate sequentially or in parallel.

While the present subject matter has been described in detail withrespect to specific example embodiments thereof, it will be appreciatedthat those skilled in the art, upon attaining an understanding of theforegoing may readily produce alterations to, variations of, andequivalents to such embodiments. Accordingly, the scope of the presentdisclosure is by way of example rather than by way of limitation, andthe subject disclosure does not preclude inclusion of suchmodifications, variations and/or additions to the present subject matteras would be readily apparent to one of ordinary skill in the art.

What is claimed is:
 1. A computer-implemented method of creatinghigh-dynamic range images, the method comprising: determining, by one ormore computing devices, a signal-to-noise ratio objective associatedwith an imaging platform configured to capture one or more images of aregion of interest, the signal-to-noise ratio objective specifying adesired signal-to-noise ratio behavior as a function of a brightness ofthe region of interest; determining, by the one or more computingdevices, an exposure profile associated with the imaging platform basedat least in part on the signal-to-noise ratio objective; obtaining, bythe one or more computing devices, data indicative of a plurality ofimage frames, each depicting at least a portion of the region ofinterest, the plurality of image frames being captured by the imagingplatform in accordance with the exposure profile; and generating, by theone or more computing devices, a high-dynamic range image of at least aportion of the region of interest based at least in part on theplurality of image frames.
 2. The computer-implemented method of claim1, wherein the exposure profile comprises a plurality of integrationstimes to be used in capturing the plurality of image frames.
 3. Thecomputer-implemented method of claim 2, wherein the exposure profilespecifies an integration time for each image frame to be captured. 4.The computer-implemented method of claim 2, wherein the plurality ofintegration times are determined to shape a signal-to-noise ratio curvein accordance with the signal-to-noise ratio objective.
 5. Thecomputer-implemented method of claim 2, wherein each image frame of theplurality of image frames is captured using a different integration timefrom the exposure profile.
 6. The computer-implemented method of claim1, wherein capturing, by the one or more computing devices, a pluralityof image frames depicting the region of interest comprises scanning theregion of interest using a two-dimensional image sensor.
 7. Thecomputer-implemented method of claim 5, wherein the two-dimensionalimage sensor is a two-dimensional staring sensor configured to capture atwo-dimensional image frame in a single snapshot.
 8. Thecomputer-implemented method of claim 6, wherein the two-dimensionalstaring sensor comprises a complementary metal-oxide-semiconductorsensor, a charge coupled device sensor, or an active-pixel sensor. 9.The computer-implemented method of claim 1, wherein the signal-to-noiseratio objective is to maintain a signal-to-noise ratio curve within apredetermined threshold range.
 10. The computer-implemented method ofclaim 1, further comprising determining, by the one or more computingdevices, a brightness distribution associated with the region ofinterest; and wherein the signal-to-noise ratio objective is to shape asignal-to-noise ratio curve based at least in part on the brightnessdistribution.
 11. The computer-implemented method of claim 10, whereindetermining, by the one or more computing devices, a brightnessdistribution associated with the region of interest comprisesdetermining a brightness distribution based at least in part on alocation of the imaging platform, a location of the sun relative to theearth, an albedo associated with the region of interest, atmosphericabsorption, an expected reflectance of one or more targets within theregion of interest, or expected atmospheric conditions.
 12. Thecomputer-implemented method of claim 11, wherein the brightnessdistribution is determined prior to capturing the plurality of imageframes.
 13. The computer-implemented method of claim 10, whereindetermining, by the one or more computing devices, a brightnessdistribution associated with the region of interest comprisesdetermining a brightness distribution associated with at least one imageframe of the plurality of image frames.
 14. A computing system,comprising: one or more processors; and one or more memory devices, theone or more memory devices storing computer-readable instructions thatwhen executed by the one or more processors cause the one or moreprocessors to perform operations, the operations comprising: determininga signal-to-noise ratio objective associated with an imaging platformconfigured to capture one or more images of a region of interest, thesignal-to-noise ratio objective specifying a desired signal-to-noiseratio curve as a function of a brightness of the region of interest;determining an exposure profile associated with the imaging platformbased at least in part on the signal-to-noise ratio objective; obtainingdata indicative of a plurality of image frames, each depicting at leasta portion of the region of interest, the plurality of image frames beingcaptured by the imaging platform in accordance with the exposureprofile; and generating a high-dynamic range image of at least a portionof the region of interest based at least in part on the plurality ofimage frames.
 15. The computing system of claim 14, further comprisingdetermining a brightness distribution associated with the region ofinterest; and wherein the signal-to-noise ratio objective is to shapethe signal-to-noise ratio curve based at least in part on the brightnessdistribution.
 16. The computing system of claim 15, wherein thebrightness distribution is determined based at least in part on alocation of the imaging platform, a location of the sun relative to theearth, an expected reflectance of one or more targets within the regionof interest, or expected atmospheric conditions.
 17. The computingsystem of claim 14, wherein the exposure profile comprises a pluralityof integrations times to be used in capturing the plurality of imageframes.
 18. One or more tangible, non-transitory computer-readable mediastoring computer-readable instructions that when executed by one or moreprocessors cause the one or more processors to perform operations, theoperations comprising: determining a signal-to-noise ratio objectiveassociated with an imaging platform configured to capture one or moreimages of a region of interest, the signal-to-noise ratio objectivespecifying a desired signal-to-noise ratio curve as a function of abrightness of the region of interest; determining an exposure profileassociated with the imaging platform based at least in part on thesignal-to-noise ratio objective; obtaining data indicative of aplurality of image frames, each depicting at least a portion of theregion of interest, the plurality of image frames being captured by theimaging platform in accordance with the exposure profile; and generatinga high-dynamic range image of at least a portion of the region ofinterest based at least in part on the plurality of image frames. 19.The one or more tangible, non-transitory computer-readable media ofclaim 18, the operations further comprising determining a brightnessdistribution associated with the region of interest; and wherein thesignal-to-noise ratio objective is to shape the signal-to-noise ratiocurve based at least in part on the brightness distribution.
 20. The oneor more tangible, non-transitory computer-readable media of claim 19,wherein the brightness distribution is determined based at least in parton a location of the imaging platform, a location of the sun relative tothe earth, an expected reflectance of one or more targets within theregion of interest, or expected atmospheric conditions.